I served for decades. I am still in. My experiences are as follows:
1. I was doing the TMS, on Judicial Committees, Circuit Assignments, etc. I gave hundreds of talks/parts, etc over the years. I became legitimately sick due to the stress of the work and aging. I told the BOE. They asked me to stay. I said, Okay but I cant do anything for a period of time. I have to catch my breath. They agreed. At the next Meeting I was asked to do a part on short notice. So much for BOE sympathy.
2. I decided to move my residence to a new town, house, etc. I moved with a good letter of recommendation. I told the Presiding Overseer I would not be serving in the new Congregation for health reasons (legitimate). My wife was good with it. He still doesn't talk to me (I really dont care). I am still there, and the freedom is liberating and recuperative to say the least. The rest of the BOE and the Cong are good to us. All is fine.
3. I can keep going to Meetings/CA/SAD/DC and in service under this arrangement. I just sit there and watch, comment once in a while that's it. I am not in the TMS (what's the point?). No jobs to do, responsibility, COs to deal with, problems to handle. Very comfortable.
4. The various COs try and talk to me to reinlist me, but I slip out the back door quickly enough.
My suggestion is do what is best for you and your family. Just make sure your wife is in agreement with it and can handle it. If you decide to withdraw she may get emotional when the announcement is made. Measure twice and cut once. Once you cut this role out of your life, and if decide you want to do it again, it may take a long while for it to happen.